Papers with generating natural language descriptions
Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning (2024.findings-acl)
Copied to clipboard
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji
| Challenge: | LVLMs are known for producing text that is factually inconsistent with visual input . factuality of generated captions for structured visuals has not been studied as much . |
| Approach: | They propose a typology of factual errors in captions generated by large vision-language models . they propose CHOCOLATE, a visual entailment model that outperforms current models based on this analysis . |
| Outcome: | The proposed model outperforms current models in evaluating caption factuality. |
Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization (N18-1)
Copied to clipboard
| Challenge: | a proposed model for generating natural language descriptions is too generic and does not exploit task specific characteristics. |
| Approach: | They propose a model which uses a fused bifocal attention mechanism to exploit micro and macro level information and a gated orthogonalization mechanism to ensure that a field is remembered for a few time steps and then forgotten. |
| Outcome: | The proposed model improves on a recently released dataset with two similar datasets for French and German. |
HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning (2026.findings-eacl)
Copied to clipboard
| Challenge: | haptic captioning is the task of generating natural language descriptions from haptics, such as vibrations, for use in virtual reality and rehabilitation applications. |
| Approach: | They propose a multimodal sensory language model that interprets vibration signals into descriptions in a given sensory, emotional, or associative category. |
| Outcome: | The proposed model interprets vibration signals into descriptions in a given sensory, emotional, or associative category. |
Hallucination Mitigation in Natural Language Generation from Large-Scale Open-Domain Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Graph-to-text models trained on small-scale datasets or datasets with limited variety of graph shapes are not adequate for more realistic large-scale, open-domain settings. |
| Approach: | They propose a novel approach that, given a graph-sentence pair in GraphNarrative, trims the sentence to eliminate portions that are not present in the corresponding graph. |
| Outcome: | The proposed model can be trained on existing datasets and is available on github. |